Skip to content

fuse: re-validate the DLM grant after waiting on the coherency gate#191

Open
hbirth wants to merge 1 commit into
DDNStorage:redfs-ubuntu-hwe-6.17.0-16.16-24.04.1from
hbirth:redfs-ubuntu-hwe-6.17.0-16.16-24.04.1
Open

fuse: re-validate the DLM grant after waiting on the coherency gate#191
hbirth wants to merge 1 commit into
DDNStorage:redfs-ubuntu-hwe-6.17.0-16.16-24.04.1from
hbirth:redfs-ubuntu-hwe-6.17.0-16.16-24.04.1

Conversation

@hbirth

@hbirth hbirth commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Both cached IO paths request their DLM lock first and then go to sleep on things a NOTIFY invalidate can be holding: the read path blocks on the coherency gate (writer priority), the write path additionally sleeps on a contended i_rwsem. A NOTIFY invalidate running in that window revokes exactly the lock just granted (fuse_dlm_unlock_range()), so the task wakes up and populates or dirties the page cache with no DLM coverage.

Close the window without ever sending a FUSE_DLM_WB_LOCK request while holding the gate (a grant that had to wait on an invalidate delivered to this same client would deadlock against our own gate hold):

  • Drop the lock record under the gate write side in fuse_reverse_inval_inode(), so revocation and page drop are one atomic step with respect to the gate.

  • After entering the gate read side, re-check the grant against the live lock tree; if it was revoked while we waited, drop the gate, re-request, re-enter and check again. With the revoke now gated, passing the check means the lock cannot go away for the whole gate hold: a revoke arriving mid-operation parks until the IO is done.

  • Move the write path's lock request below the inode lock, so the sleep on i_rwsem no longer sits between grant and use. This also computes the O_APPEND lock range against a stable i_size instead of a racy lockless read.

fuse_get_dlm_lock() now reports whether the grant is recorded so the re-validation loop can fall through unlocked on a persistent request failure, preserving the old degraded behavior instead of spinning.

Both cached IO paths request their DLM lock first and then go to sleep
on things a NOTIFY invalidate can be holding: the read path blocks on
the coherency gate (writer priority), the write path additionally
sleeps on a contended i_rwsem.  A NOTIFY invalidate running in that
window revokes exactly the lock just granted (fuse_dlm_unlock_range()),
so the task wakes up and populates or dirties the page cache with no
DLM coverage.

Close the window without ever sending a FUSE_DLM_WB_LOCK request while
holding the gate (a grant that had to wait on an invalidate delivered
to this same client would deadlock against our own gate hold):

 - Drop the lock record under the gate write side in
   fuse_reverse_inval_inode(), so revocation and page drop are one
   atomic step with respect to the gate.

 - After entering the gate read side, re-check the grant against the
   live lock tree; if it was revoked while we waited, drop the gate,
   re-request, re-enter and check again.  With the revoke now gated,
   passing the check means the lock cannot go away for the whole gate
   hold: a revoke arriving mid-operation parks until the IO is done.

 - Move the write path's lock request below the inode lock, so the
   sleep on i_rwsem no longer sits between grant and use.  This also
   computes the O_APPEND lock range against a stable i_size instead of
   a racy lockless read.

fuse_get_dlm_lock() now reports whether the grant is recorded so the
re-validation loop can fall through unlocked on a persistent request
failure, preserving the old degraded behavior instead of spinning.

Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
@hbirth

hbirth commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

@achhenderson @hazhou-ddn the key was to create a way to query whether we have a region marked as exclusive (this was not complete before since we never needed it) ... now we can operate in a loop ... if we lost the dlm lock we can reacquire it without holding the semaphore. This still has a small problem, where we probably acquire a lock twice when two writer were blocked by the notification ... but I think our code can handle that

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant